Meditations on Visual Quality

Digital video is one of the most attractive meansof conveying information, and the quality of thevideo is one of the determining factors for theend user experience. In order to improve thevisual user experience, one has to understand thequality expectation of users for the specific videoapplication, and deliver upon good video qualityor find ways to enhance it. The objective of this paper is to present various methods for videoquality enhancement. We will review differentmethods available first from the encoder side andsecondly from the decoder side. CAUSES OF VIDEO ARTIFACTS Video quality can be influenced from variousfactors starting from video capturing process,ending in display. The artifacts can be introducedat various stages of an end-to-end video system,including pre-processing, compression,transmission, post-processing and display. Thevideo source can be noisy due to sensorcharacteristics of capturing device whether it is asingle or multi-camera. Various compressionartifacts such as blocking, blurring, ringing,color bleeding, flickering can be produced duringencoding [1]. Video compression artifacts arevaried by the available bit budget and theperformance of the encoding algorithms. Inaddition to compression, quality can degradeseverely due to possible losses during transmission. Tolerance to artifacts andacceptance of quality can vary depending on thedevice (whether it is a mobile phone or 3DTV)and the environment which the video isconsumed, and lastly the expectation of user. In order to improve the quality of video, theaforementioned artifacts have to be eitherreduced if not eliminated during encodingprocess or should be processed and masked afterdecoding. In the next sections we will addressthese two approaches. MORE BITS: ENCODER SIDE Video quality can be enhanced from the encoderside, simply by spending extra bits duringcompression. Spending more bits duringcompression gives the flexibility to encoder toprovide better quality without increasing thecomplexity. On the other hand, encoder has thechoice to enhance the quality by using morecomplex algorithms such as better motionestimation, better rate control and predictionschemes. However, even for very high-bit ratevideo compression applications such as HDvideo, where the bit-budget is very high, thegiven bit-budget has to be appropriatelydistributed among different scenes of a sourcevideo as well as on different regions of a frame.At the video sequence level, distributing the bits among different frames requires a good multi-pass rate control algorithm [2]. At the frame level, allocating bits to different regions of avideo frame can be based on techniques such asROI (region of interest) identification [3], and byusing perceptual cues [4]. In the case of low bit rate video compression,one can send video content at full frame rate (30fps) where the quality will be degraded byannoying blocking artifacts, due to the limited bitbudget. In order to avoid compression artifacts,the frame rate may be reduced to 15 or 10 framesper second (fps) where each frame is encodedwith more bits and therefore has better spatialvideo quality [5] as shown in the example inFigure 1. However in this case, a recoverymechanism utilizing frame rate up conversion isneeded at the decoder to display the video at ahigher frame rate [6], otherwise the video willsuffer from motion jerkiness due to low framerate. In another scenario, the video can be sent atfull frame rate but at reduced resolution. Thisrequires a good image down-sampling algorithm